{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  28.24414088197752\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.294\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  15.396240321444028\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.384\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  4.200594650104354\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.706\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.8928964971078328e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.3025877108306494\n",
      "W1 relative error: 1.34e-07\n",
      "W2 relative error: 8.28e-08\n",
      "W3 relative error: 2.40e-06\n",
      "b1 relative error: 1.40e-08\n",
      "b2 relative error: 1.00e+00\n",
      "b3 relative error: 1.10e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3090066830824556\n",
      "W1 relative error: 3.10e-07\n",
      "W2 relative error: 6.65e-08\n",
      "W3 relative error: 1.14e-08\n",
      "b1 relative error: 1.54e-08\n",
      "b2 relative error: 9.20e-10\n",
      "b3 relative error: 9.88e-11\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.880000; val_acc: 0.268000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.277000\n",
      "(Epoch 10 / 25) train acc: 0.898000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.924000; val_acc: 0.263000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.300000\n",
      "(Epoch 13 / 25) train acc: 0.972000; val_acc: 0.314000\n",
      "(Epoch 14 / 25) train acc: 0.972000; val_acc: 0.310000\n",
      "(Epoch 15 / 25) train acc: 0.974000; val_acc: 0.314000\n",
      "(Epoch 16 / 25) train acc: 0.994000; val_acc: 0.304000\n",
      "(Epoch 17 / 25) train acc: 0.970000; val_acc: 0.305000\n",
      "(Epoch 18 / 25) train acc: 0.990000; val_acc: 0.311000\n",
      "(Epoch 19 / 25) train acc: 0.988000; val_acc: 0.308000\n",
      "(Epoch 20 / 25) train acc: 0.992000; val_acc: 0.287000\n",
      "(Iteration 101 / 125) loss: 0.001417\n",
      "(Epoch 21 / 25) train acc: 0.994000; val_acc: 0.291000\n",
      "(Epoch 22 / 25) train acc: 0.998000; val_acc: 0.308000\n",
      "(Epoch 23 / 25) train acc: 0.996000; val_acc: 0.308000\n",
      "(Epoch 24 / 25) train acc: 0.998000; val_acc: 0.307000\n",
      "(Epoch 25 / 25) train acc: 0.994000; val_acc: 0.305000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 35.055681\n",
      "(Epoch 0 / 25) train acc: 0.250000; val_acc: 0.196000\n",
      "(Epoch 1 / 25) train acc: 0.398000; val_acc: 0.240000\n",
      "(Epoch 2 / 25) train acc: 0.426000; val_acc: 0.237000\n",
      "(Epoch 3 / 25) train acc: 0.496000; val_acc: 0.258000\n",
      "(Epoch 4 / 25) train acc: 0.540000; val_acc: 0.247000\n",
      "(Epoch 5 / 25) train acc: 0.608000; val_acc: 0.295000\n",
      "(Epoch 6 / 25) train acc: 0.596000; val_acc: 0.258000\n",
      "(Epoch 7 / 25) train acc: 0.710000; val_acc: 0.276000\n",
      "(Epoch 8 / 25) train acc: 0.740000; val_acc: 0.303000\n",
      "(Epoch 9 / 25) train acc: 0.760000; val_acc: 0.281000\n",
      "(Epoch 10 / 25) train acc: 0.802000; val_acc: 0.280000\n",
      "(Epoch 11 / 25) train acc: 0.842000; val_acc: 0.309000\n",
      "(Epoch 12 / 25) train acc: 0.822000; val_acc: 0.297000\n",
      "(Epoch 13 / 25) train acc: 0.836000; val_acc: 0.275000\n",
      "(Epoch 14 / 25) train acc: 0.886000; val_acc: 0.277000\n",
      "(Epoch 15 / 25) train acc: 0.872000; val_acc: 0.296000\n",
      "(Epoch 16 / 25) train acc: 0.886000; val_acc: 0.296000\n",
      "(Epoch 17 / 25) train acc: 0.920000; val_acc: 0.298000\n",
      "(Epoch 18 / 25) train acc: 0.914000; val_acc: 0.291000\n",
      "(Epoch 19 / 25) train acc: 0.938000; val_acc: 0.303000\n",
      "(Epoch 20 / 25) train acc: 0.956000; val_acc: 0.314000\n",
      "(Iteration 101 / 125) loss: 2.094676\n",
      "(Epoch 21 / 25) train acc: 0.958000; val_acc: 0.288000\n",
      "(Epoch 22 / 25) train acc: 0.962000; val_acc: 0.280000\n",
      "(Epoch 23 / 25) train acc: 0.972000; val_acc: 0.292000\n",
      "(Epoch 24 / 25) train acc: 0.980000; val_acc: 0.279000\n",
      "(Epoch 25 / 25) train acc: 0.948000; val_acc: 0.287000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA34AAAJNCAYAAABusKejAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde5hcZZXo/++i6UgPYFolXtIdTRQMKonEafASbyNHgxckoEbgKKLjMKCcOP7OyQzMc2T4ZcYhTuaI8gNlELzAMGJmDBFGNM4ELwNeSIdgh4s5RETSHdSAdkRsJAnr90dVQ6XpbtLpqtpdVd/P8+Sp3u/etffa3dWVWv2+73ojM5EkSZIkNa/9ig5AkiRJklRbJn6SJEmS1ORM/CRJkiSpyZn4SZIkSVKTM/GTJEmSpCZn4idJkiRJTc7ET5LUMiKiLSJ+FxHPLToWSZLqKVzHT5I0VUXE7yo2/wj4A7C7vP3nmXlV/aOSJKnxmPhJkhpCRNwDfDAz/3OcY/bPzF31i6p+mvneJEm151BPSVLDioi/i4ivRMSXI+JB4D0R8cqI+GFEDEbEfRFxYUS0l4/fPyIyImaXt/+5vP8bEfFgRPwgIuaMca39IuLfIuIX5XN/JyJeVLH/jyLigoi4NyJ2RMT3IuIp5X2vLce0IyK2RsR7y+03RsRpFef4YER8Z0SsH4qILcBPyu0XRUR/RPw2ItZHxKsqnr9/RHwsIn5a3t8bETMj4p8i4hMj7ucbEXHW5H8KkqRGYOInSWp0JwD/AkwHvgLsAj4CHAIsBI4F/nyc558CfAx4OnAv8LfjHPvvwGHAs4HbgCsr9l0AzAdeXj7XXwOPlhPJrwOfBJ4BLAA2TeD+3g4cBcwrb/+ofJ2nA/8G/OtwggksA95J6Z47gQ8CDwNfAk6JiP0AIuJZwOuAqycQhySpgZn4SZIa3Y2ZeV1mPpqZQ5m5PjN/lJm7MvNu4FJKSc5Y/i0zezNzJ3AVcORoB5XP/8XMfDAzHwbOA/44Ig6MiDbgNGBpZt6Xmbsz88byOd8DfDMzV5Vjuj8zb53A/f19Zv4mM4fKcVyZmb8uD/v8B+CpwKHlYz8I/HVm3lWO99bysd8Hhiq+DycD/5mZ908gDklSAzPxkyQ1uq2VGxFxeER8vTwk87fAckq9f2P5RcXXvwcOGu2gckXQf4iIu8vn3VLedQjwLGAa8NNRnjprjPa9NfL+/jIifhIRO4DfAAfy+P2Nd60rKCWhlB+vHOM4SVITMvGTJDW6kVXK/onSMMxDM/OpwLlAVOE6pwJvAd5AaVjpcC9bAL8EHgFeMMrzto7RDvAQpWqlw549yjGP3V9E/Anw/wDvoDSU82nA73j8/sa71pXAiRGxoHzMdWMcJ0lqQiZ+kqRmczCwA3ioXHxlvPl9Ez3vH4AHKCVrHx/ekZm7gS8Cn4qIZ5d7BxeWi8r8M3BsRLyjXHzlkIh4afmptwLviIiOiHgh8IG9iGEXcD/QTmm46YEV+y8D/i4iXhAlR0bE08sx/rx8vS8B/1oeripJahEmfpKkZvM/gfcBD1Lq/ftKlc77BWBb+d/twPdH7P8ocCewAfg18PeUlk36GXAc8Ffl9lt4vFDLP1Lq0fsV8HlKSeJ4rgf+E7gLuAf4LXBfxf6VwBpgXXnfpcABFfu/VL62wzwlqcW4jp8kSS0iIt4AXA48P/0AIEktxR4/SZJaQERMo7TMxedM+iSp9Zj4SZLU5CJiHqUKoE8HLiw4HElSARzqKUmSJElNzh4/SZIkSWpyJn6SJEmS1OT2LzqAajrkkENy9uzZRYchSZIkSYXYsGHD/Zk5Y2R7UyV+s2fPpre3t+gwJEmSJKkQEfHz0dod6ilJkiRJTc7ET5IkSZKanImfJEmSJDU5Ez9JkiRJanImfpIkSZLU5ApJ/CLi8xHxq4i4bYz9EREXRsSWiOiLiJfVO0ZJkiRJahZFLefwReAi4Iox9r8ZOKz87+XAZ8uPkiRJkiqs2TjAyrWb2TY4xMzODpYtmsviBV1Fh1VTrXjPk1VI4peZ34uI2eMccjxwRWYm8MOI6IyI52TmfXUJUJIkNSU/LKqWinh9rdk4wDmrNzG0czcAA4NDnLN6E0BdXtvec+O8j0zVBdy7gK0V2/3lNhM/SZK0T4r+sFiERv2AOhlF3XNRr6+Vazc/ds1hQzt3s3Lt5qZNwFrxnqthqhZ3iVHactQDI06PiN6I6N2+fXuNw5IkSY1qvA+LzWj4A+rA4BDJ4x9Q12wcqNv1F664gTlnf52FK26oy3WLvOeiXl/bBocm1F5N3nNJo7yPTNXErx+YVbHdDWwb7cDMvDQzezKzZ8aMGXUJTpIkNZ4iPywWocgPqEUlYEXec1Gvr5mdHRNqrybv+cnbp5KpmvhdC5xaru75CmCH8/skSdJkFPlhsQit2CtS5D0X9fpatmguHe1te7R1tLexbNHcml4Xir3nd077PjdOW8rdTzmFG6ct5Z3Tvt/U91wNRS3n8GXgB8DciOiPiD+NiDMi4ozyIdcDdwNbgM8BHyoiTkmS1DyK/IBchFbsFSnynot6fS1e0MX5J86jq7ODALo6Ozj/xHl1mW9W2D233cSK9svo3u9+9gvo3u9+VrRfxuK2m2p6XWjs95Giqnqe/CT7E/hwncKRJEl1VFTxjeFrtEqxk2WL5u5RhALq2xM0MEqSV4+eoKLuucjX1+IFXYW8jgu753XL2X/3w3s07b/7YVi3HOYvqemlG/l9JEo5VnPo6enJ3t7eosOQJGmvFVl1cSqUYYfSB/N69VAUpchKk1OhwiXU7+fcipVMC9W3qpRw7eiH6d1wzLk1T744r5PR6z4GnDdY22s3gIjYkJk9T2g38ZMkqRhFfzgu4toLV9wwak9QV2cHN539hppdt0gmuyZgTatvFVy3FHZW/E63d8BxF9Y2+bvgCNix9Ynt02fBR2+r3XUbxFiJ31Qt7iJJUtMrsgJhKxbfKEojl3+fjMULurjp7DfwsxVv5aaz32DS14zWLd8z6YPS9rrltb3uMeeWEsxK7R2ldo1pqi7gLklSXRXRO1FkElRk8Y0i5n4VqRWTXdVZEcMtoXS9ibRXy/C9FXHPDczET5LU8kYOxRtecwyoafJXZBLUisU3itKKyW6hikqCirr2yOGWO7aWtqH2157ePcaQy+7aXhdK92aiNyEO9ZQktbyihuIVWRa8FUvPF6WRy783nOEkaMdWIB9PgvpWNe+1ixpuCQ65bDD2+EmSWl5RQ/GKLv9e5LWbOdEbqZHLvzec8ZKgWvcOFXXtooZbgkMuG4yJnySp5RU5FK/IJKjVErAi+b2ukyKToKKuXeRwS2jNIZdFDieeBId6SpJankPxpCYxVrJTjySoqGs73LK+ihxOPEkmfpKklteK886kplRkElTUtecvKa2bN30WEKXHWq+j18qKnFM5SQ71lCQJh+JJVVfEcLgi55wVfW0TvfoocjjxJJn4SZIkqbqKXGKgyCTIBKz5FT2nchIc6ilJkqTqauDhcNK4GnhOpYmfJEmSqquBh8NJ42rgOZUO9ZQkTRlrNg641pnUDBp4OJz0pBp0SK89fpKkKWHNxgHOWb2JgcEhEhgYHOKc1ZtYs3Gg6NCkyetbBRccAed1lh4boPT7pDTwcDipWZn4SZKmhJVrNzO0c/cebUM7d7Ny7eaCIpKqpIHX/dpnDTwcTmpWDvWUJE0J2waHJtQuNYzxCp3UOhEqYkmFYQ06HE5qVvb4SZKmhJmdHRNqlxpGUYVOWrGnUdKYTPwkSVPCskVz6Whv26Oto72NZYvmFhSRVCVjFTSpdaETl1SQVKGQxC8ijo2IzRGxJSLOHmX/0yLimojoi4ibI+KIIuKUpKKt2TjAwhU3MOfsr7NwxQ1NXehk8YIuzj9xHl2dHQTQ1dnB+SfOs6qnGl9RhU5cUkFShbrP8YuINuBi4I1AP7A+Iq7NzDsqDvtr4NbMPCEiDi8ff0y9Y5WkIg1XuRwueDJc5RKoeTJU1LIKixd0meip+QzPc6v3XDuXVJBUoYjiLkcDWzLzboCIuBo4HqhM/F4MnA+QmT+JiNkR8azM/GXdo5WkgoxX5bKWyVGRCafUtIoodHLMuaU5fZXDPV1SQWpZRQz17AIq//zUX26r9GPgRICIOBp4HuCfpyS1lKKqXLqsgtQkXFJBUoUievxilLYcsb0C+HRE3ApsAjYCu0Y9WcTpwOkAz33uc6sYpiQVa2ZnBwOjJHm1rnLpsgpSE3FJBUllRfT49QOzKra7gW2VB2TmbzPz/Zl5JHAqMAP42Wgny8xLM7MnM3tmzJhRq5glqe6KqnLpsgotpG8VXHAEnNdZemyFMv+teM+SRDGJ33rgsIiYExHTgJOAaysPiIjO8j6ADwLfy8zf1jlOSQKKq6xZVJVLl1VoEa24xlsr3rMklUXmyFGWdbhoxFuATwFtwOcz8+MRcQZAZl4SEa8ErgB2Uyr68qeZ+ZsnO29PT0/29vbWMHJJrWZkoRMoJUHNvsxAUVU9VUcXHDFGxcdZ8NHb6h9PPbTiPUtqORGxITN7RrYXMcePzLweuH5E2yUVX/8AOKzecUnSSEVV1iyayyq0gFZc460V71mSygpZwF2SGoWFTtS0xlrLrR5rvBU1z67Ie5akgpn4SdI4LHSipnXMuaU13SrVY423IufZFXXPkjQFmPhJ0jgsdKKmVdQab+uW77mgOJS21y2v7XXBde0ktbRC5vhJUqMYnudmoRM1pSLWeCt6np3r2klqUSZ+kvQkLHQiVdH07jEqazrPTpJqyaGekiSpfpxnJ0mFMPGTJEn14zw7SSqEQz0lSVJ9Oc9OkurOHj9JkiRJanImfpKkqaOohb0lSWpyDvWUJE0Nwwt7D6/xNrywNzT3sMC+VaU17Hb0lypbHnNuc9+vJKkQJn6SGsaajQOup9fMxlvYu1kToVZNdiVJdedQT0kNYc3GAc5ZvYmBwSESGBgc4pzVm1izcaDo0FQtRS/sXYTxkl1JkqrIxE9SQ1i5djNDO3fv0Ta0czcr124uKCJV3VgLeDfzwt6tmOxKkgph4iepIWwbHJpQuxpQ0Qt7F1FYphWTXUlSIUz8JDWEmZ0dE2pXAypyYe/huXY7tgL5+Fy7Wid/RSe7kqSWYXEXSQ1h2aK5nLN60x7DPTva21i2aG6BUanqilrYu6jCMsPntqqnJKnGTPwkNYTh6p1W9VRNFDnXrqhkV5LUUkz8JDWMxQu6TPRUG9O7y8M8R2mXJKkJOMdPkiTn2kmSmlwhiV9EHBsRmyNiS0ScPcr+6RFxXUT8OCJuj4j3FxGnJLWsIipcFqnIwjKSJNVB3Yd6RkQbcDHwRqAfWB8R12bmHRWHfRi4IzOPi4gZwOaIuCozH6l3vJLUcoYrXA4XOxmucAnNnQg5106S1MSK6PE7GtiSmXeXE7mrgeNHHJPAwRERwEHAr4Fd9Q1TklrUeBUuJUlSQyoi8esCKmfQ95fbKl0EvAjYBmwCPpKZj9YnPElqcUVWuJQkSTVRROIXo7TliO1FwK3ATOBI4KKIeOqoJ4s4PSJ6I6J3+/bt1Y1UklrRWJUsrXApSVLDKiLx6wdmVWx3U+rZq/R+YHWWbAF+Bhw+2sky89LM7MnMnhkzZtQkYElqKVa4lCSp6RSR+K0HDouIORExDTgJuHbEMfcCxwBExLOAucDddY1SklqVFS4lSWo6da/qmZm7IuIsYC3QBnw+M2+PiDPK+y8B/hb4YkRsojQ09K8y8/56xypJLcsKl5IkNZW6J34AmXk9cP2Itksqvt4GvKnecUmSJElSMypkAXdJkiRJUv2Y+EmSJElSkzPxkyRJkqQmZ+InSVNZ3yq44Ag4r7P02Leq6IgkSVIDKqS4iyRpL/StguuWws6h0vaOraVtsOKmJEmaEHv8JGmqWrf88aRv2M6hUrskSdIEmPhJ0lS1o39i7ZIkSWMw8ZOkqWp698TaJUmSxmDiJ0lT1THnQnvHnm3tHaV2SZKkCTDxk6QnU1RlzflL4LgLYfosIEqPx11oYRdJkjRhVvWUpPEUXVlz/hITPUmSNGn2+EnSeKysKUmSmoCJnySNx8qakiSpCZj4SdJ4rKwpSZKagImfJI3HypqSJKkJWNxF0oSs2TjAyrWb2TY4xMzODpYtmsviBV1Fh1U7w4VV1i0vDe+c3l1K+iy4IkmSGoiJn6S9tmbjAOes3sTQzt0ADAwOcc7qTQDNn/yZ6EmSpAbmUE9Je23l2s2PJX3DhnbuZuXazQVFJEmSpL1h4idpr20bHJpQe9UVtZC6JElSg3Oop9SgiphrN7Ozg4FRkryZnR2jHF1lRS+kLkmS1MAK6fGLiGMjYnNEbImIs0fZvywibi3/uy0idkfE04uIVZqKhufaDQwOkTw+127NxoGaXnfZorl0tLft0dbR3sayRXNrel3AhdQlSZImoe6JX0S0ARcDbwZeDJwcES+uPCYzV2bmkZl5JHAO8N3M/HW9Y5WmqqLm2i1e0MX5J86jq7ODALo6Ozj/xHn1KeziQuqSJEn7rIihnkcDWzLzboCIuBo4HrhjjONPBr5cp9ikhlDkXLvFC7qKqeA5vbs0vHO0dkmSJI2riKGeXUDlp7f+ctsTRMQfAccCX61DXFLDGGtOXV3m2hXFhdQlSZL2WRGJX4zSlmMcexxw03jDPCPi9IjojYje7du3VyVAaaordK5dUeYvgeMuhOmzgCg9HnehhV0kSZL2wqSGekbEWcBVmfmbCTytH5hVsd0NbBvj2JN4kmGemXkpcClAT0/PWAmk1FSGh1rWu6pn4VxIXZIkaZ9Mdo7fs4H1EXEL8HlgbWY+WfK1HjgsIuYAA5SSu1NGHhQR04HXAe+ZZIxSUypsrp0kSZIazqSGembm/wYOAy4HTgPuioi/j4gXjPOcXcBZwFrgTmBVZt4eEWdExBkVh54AfCszH5pMjJIkSZLU6iZd1TMzMyJ+AfwC2AU8Dfi3iPiPzPzLMZ5zPXD9iLZLRmx/EfjiZOOTJEmSpFY32Tl+S4H3AfcDlwHLMnNnROwH3AWMmvhJkiRJkupnsj1+hwAnZubPKxsz89GIeNskzy1JkiRJqoLJLudwPfDYUgsRcXBEvBwgM++c5LklSZIkSVUw2cTvs8DvKrYfKrdJkiRJkqaIySZ+Ubl8Q2Y+ShUKxkiSJEmSqmeyid/dEbE0ItrL/z4C3F2NwCRJkiRJ1THZxO8M4FWUFmLvB14OnD7ZoCRJkiRJ1TOpYZmZ+SvgpCrFIkmSJEmqgcmu43cA8KfAS4ADhtsz8wOTjEuSJEmSVCWTHep5JfBsYBHwXaAbeHCyQUmSJEmSqmeyid+hmfkx4KHM/BLwVmDe5MOSJEmSJFXLZBO/neXHwYg4ApgOzJ7kOSVJkiRJVTTZNfcujYinAf8buBY4CPjYpKOSJEmSJFXNPid+EbEf8NvM/A3wPeD5VYtKkiRJklQ1+zzUMzMfBc6qYiySJEmSpBqY7By//4iI/xURsyLi6cP/qhKZJEmSJKkqJjvHb3i9vg9XtCUO+5QkSZKkKWNSiV9mzqlWIJIkSZKk2phU4hcRp47WnplXTOa8kiRJkqTqmexQz6Mqvj4AOAa4BTDxk2qtbxWsWw47+mF6NxxzLsxfUnRUkiRJmoImO9Tzf1RuR8R04Mone15EHAt8GmgDLsvMFaMc83rgU0A7cH9mvm4ysUpNpW8VXLcUdg6VtndsLW2DyZ8kSZKeYLI9fiP9HjhsvAMiog24GHgj0A+sj4hrM/OOimM6gc8Ax2bmvRHxzCrHKVXFmo0DrFy7mW2DQ8zs7GDZorksXtBV+wuvW/540jds51Cp3cRPkiRJI0x2jt91lKp4QmlpiBcDq57kaUcDWzLz7vI5rgaOB+6oOOYUYHVm3guQmb+aTJxSLazZOMA5qzcxtHM3AAODQ5yzehNA7ZO/Hf0Ta68mh5hKkiQ1nMn2+P1jxde7gJ9n5pN98uwCtlZs9wMvH3HMC4H2iPgOcDDwaQvGaKpZuXbzY0nfsKGdu1m5dnPtE7/p3aXhnaO115JDTCVJkhrSZBdwvxf4UWZ+NzNvAh6IiNlP8pwYpS1HbO8P/DHwVmAR8LGIeOGoJ4s4PSJ6I6J3+/btEwpemoxtg0MTaq+qY86F9o4929o7Su21NN4QU0mSJE1Zk038/hV4tGJ7d7ltPP3ArIrtbmDbKMd8MzMfysz7ge8BLx3tZJl5aWb2ZGbPjBkzJhS8NBkzOzsm1F5V85fAcRfC9FlAlB6Pu7D2vW5FDjGVJEnSPpvsUM/9M/OR4Y3MfCQipj3Jc9YDh0XEHGAAOInSnL5KXwMuioj9gWmUhoJeMMlYpapatmjuHnP8ADra21i2aG59Api/pP7DK4saYipJkqRJmWyP3/aIePvwRkQcD9w/3hMycxdwFrAWuBNYlZm3R8QZEXFG+Zg7gW8CfcDNlJZ8uG2SsUpVtXhBF+efOI+uzg4C6Ors4PwT59WnqmdRihpiKkmSpEmJzJHT6ybw5IgXAFcBM8tN/cCpmbmlCrFNWE9PT/b29hZxaal1WNVTkiRpyoqIDZnZM7J9sgu4/xR4RUQcRCmJfHAy55PUAIoYYipJkqRJmdRQz4j4+4jozMzfZeaDEfG0iPi7agUnSZIkSZq8yc7xe3NmDg5vZOZvgLdM8pySJEmSpCqabOLXFhFPGd6IiA7gKeMcL0mSJEmqs8ku5/DPwLqI+EJ5+/3AlyZ5TkmSJElSFU22uMs/REQf8N+AoLQEw/OqEZgkSZIkqTomO9QT4BfAo8A7gGMorc0nSZIkSZoi9qnHLyJeCJwEnAw8AHyF0nIOf1LF2CRJkiRJVbCvQz1/AvwXcNzwYu0R8dGqRSVJkiRJqpp9Her5DkpDPL8dEZ+LiGMozfGTJEmSJE0x+5T4ZeY1mflu4HDgO8BHgWdFxGcj4k1VjE+a2vpWwQVHwHmdpce+VUVHJEmSJD3BpIq7ZOZDmXlVZr4N6AZuBc6uSmTSVNe3Cq5bCju2All6vG6pyZ8kSZKmnGpU9QQgM3+dmf+UmW+o1jmlvbVm4wALV9zAnLO/zsIVN7Bm40DtL7puOewc2rNt51CpXZIkSZpCJruAu1S4NRsHOGf1JoZ27gZgYHCIc1ZvAmDxgq7aXXhH/8TaJUmSpIJUrcdPKsrKtZsfS/qGDe3czcq1m2t74endE2uXJEmSCmLip4a3bXBoQu1Vc8y50N6xZ1t7R6ldkiRJmkJM/NTwZnZ2TKi9auYvgeMuhOmzgCg9HndhqV2SJEmaQpzjp4a3bNHcPeb4AXS0t7Fs0dzaX3z+EhM9SZIkTXkmfmp4wwVcVq7dzLbBIWZ2drBs0dzaFnaRJEmSGoiJn5rC4gVdJnqSJEnSGAqZ4xcRx0bE5ojYEhFPWPA9Il4fETsi4tbyP6tlSJIkSdI+qnuPX0S0ARcDbwT6gfURcW1m3jHi0P/KzLfVOz5JkiRJajZF9PgdDWzJzLsz8xHgauD4AuJQM+lbBRccAed1lh77VhUdkSRJkjRlFJH4dQFbK7b7y20jvTIifhwR34iIl9QnNDWkvlVw3VLYsRXI0uN1S03+JEmSpLIiEr8YpS1HbN8CPC8zXwr8f8CaMU8WcXpE9EZE7/bt26sYphrGuuWwc8Ri7TuHSu2SJEmSCkn8+oFZFdvdwLbKAzLzt5n5u/LX1wPtEXHIaCfLzEszsycze2bMmFGrmDWV7eifWLskSZLUYopYzmE9cFhEzAEGgJOAUyoPiIhnA7/MzIyIoyklqA/UPVJNyJqNA8WspTe9uzzMc5R2SZIkSfXv8cvMXcBZwFrgTmBVZt4eEWdExBnlw94J3BYRPwYuBE7KzJHDQTWFrNk4wDmrNzEwOEQCA4NDnLN6E2s2DtT+4secC+0de7a1d5TaJUmSJBHNlE/19PRkb29v0WG0pIUrbmBgcOgJ7V2dHdx09htqH0DfqtKcvh39pZ6+Y86F+Utqf11JkiRpComIDZnZM7K9iKGeakLbRkn6xmuvuvlLTPQkSZKkMRRR3EVNaGZnx4TaJUmSJNWPiZ+qYtmiuXS0t+3R1tHexrJFcwuKSJIkSdIwh3qqKoardxZS1VOSJEnSuEz8VDWLF3SZ6EmSJElTkEM9JUmSJKnJmfhJkiRJUpMz8ZMkSZKkJmfiJ0mSJElNzsRPkiRJkpqciZ8kSZIkNTkTP0mSJElqciZ+kiRJktTkTPwkSZIkqcmZ+Kl6+lbBBUfAeZ2lx75VRUckSZIkCdi/6ADUJPpWwXVLYedQaXvH1tI2wPwlxcUlSZIkyR4/Vcm65Y8nfcN2DpXaJUmSJBXKHr8mtGbjACvXbmbb4BAzOztYtmguixd01faiO/on1i5JkiSpbuzxazJrNg5wzupNDAwOkcDA4BDnrN7Emo0Dtb3w9O6JtUuSJEmqGxO/JrNy7WaGdu7eo21o525Wrt1c2wsfcy60d+zZ1t5RapckSZJUqEISv4g4NiI2R8SWiDh7nOOOiojdEfHOesbXyLYNDk2ovWrmL4HjLoTps4AoPR53oYVdJEmSpCmg7nP8IqINuBh4I9APrI+IazPzjlGO+wSwtt4xNrKZnR0MjJLkzezsGOXoKpu/xERPkiRJmoKK6PE7GtiSmXdn5iPA1cDxoxz3P4CvAr+qZ3CNbtmiubxz2ve5cdpS7n7KKdw4bSnvnPZ9li2aW3RokiRJkgpSRFXPLmBrxXY/8PLKAyKiCzgBeANwVP1Ca3yL227ibe2Xsf/uhwHojvtZ0XYZ+7e9FLA3TpIkSWpFRfT4xShtOWL7U8BfZebuUY7d82QRp0dEb0T0bt++vSoBNrR1yx9L+obtv/th19OTJEmSWlgRPX79wKyK7W5g24hjeoCrIwLgEOAtEbErM9eMPFlmXgpcCtDT0zMygWw9rqcnSZIkaYQiEr/1wGERMQcYAE4CTqk8IDPnDH8dEV8E/n20pE+jmN4NO7aO3i5JkiSpJdV9qGdm7gLOolSt805gVWbeHhFnRMQZ9Y6n6bieniRJkqQRiujxIzOvB64f0XbJGMeeVo+YmsbwcgrrlpeGd07vLiV9LrMgSZIktaxCEj/VmEWOhRcAACAASURBVOvpSZIkSapQRFVPSZIkSVIdmfhJkiRJUpMz8ZMkSZKkJmfiJ0mSJElNzsRPkiRJkpqciZ8kSZIkNTkTP0mSJElqciZ+kiRJktTkTPwkSZIkqcmZ+EmSJElSkzPxkyRJkqQmZ+InSZIkSU3OxE+SJEmSmpyJnyRJkiQ1ORM/SZIkSWpyJn6SJEmS1ORM/CRJkiSpyZn4SZIkSVKTM/GTJEmSpCZn4idJkiRJTa6QxC8ijo2IzRGxJSLOHmX/8RHRFxG3RkRvRLy6iDglSZIkqRnsX+8LRkQbcDHwRqAfWB8R12bmHRWHrQOuzcyMiPnAKuDwescqSZIkSc2giB6/o4EtmXl3Zj4CXA0cX3lAZv4uM7O8eSCQSJIkSZL2SRGJXxewtWK7v9y2h4g4ISJ+Anwd+ECdYpMkSZKkplNE4hejtD2hRy8zr8nMw4HFwN+OebKI08vzAHu3b99exTAlSZIkqTkUkfj1A7MqtruBbWMdnJnfA14QEYeMsf/SzOzJzJ4ZM2ZUN1JJkiRJagJ1L+4CrAcOi4g5wABwEnBK5QERcSjw03Jxl5cB04AH6h6pJEmSqmbnzp309/fz8MMPFx2K1PAOOOAAuru7aW9v36vj6574ZeauiDgLWAu0AZ/PzNsj4ozy/kuAdwCnRsROYAh4d0WxF0mSJDWg/v5+Dj74YGbPnk3EaLN/JO2NzOSBBx6gv7+fOXPm7NVziujxIzOvB64f0XZJxdefAD5R77gkSZJUOw8//LBJn1QFEcEznvEMJlLjpJAF3CVJktSaTPqk6pjo75KJnyRJklrGBz7wAZ75zGdyxBFHjHlMZrJ06VIOPfRQ5s+fzy233PLYvm9+85vMnTuXQw89lBUrVuzVNQ866KBJx72vvvOd7/D973+/sOtX2958/6+66irmz5/P/PnzedWrXsWPf/zjx/bNnj2befPmceSRR9LT07NX12yWn5+JnyRJklrGaaedxje/+c1xj/nGN77BXXfdxV133cWll17KmWeeCcDu3bv58Ic/zDe+8Q3uuOMOvvzlL3PHHXfsUxy7d+/ep+dNVDMlfnv7/Z8zZw7f/e536evr42Mf+xinn376Hvu//e1vc+utt9Lb2zupWOrBxK9BrNk4wMIVNzDn7K+zcMUNrNk4UHRIkiRJDaMWn6Ve+9rX8vSnP33cY772ta9x6qmnEhG84hWvYHBwkPvuu4+bb76ZQw89lOc///lMmzaNk046ia997WtPeP7PfvYzXvnKV3LUUUfxsY997LH273znO/zJn/wJp5xyCvPmzQPgk5/8JEcccQRHHHEEn/rUpwC45557OPzww3nf+97H/Pnzeec738nvf/97ANatW8eCBQuYN28eH/jAB/jDH/4AlHqy7r//fgB6e3t5/etfzz333MMll1zCBRdcwJFHHsl//dd/Tfr7NyF9q+CCI+C8ztJj36pJnW5vv/+vetWreNrTngbAK17xCvr7+yd0nWb9+Zn41ciajQOcs3oTA4NDJDAwOMQ5qzeZ/EmSJO2FIj9LDQwMMGvW48tOd3d3MzAwMGb7SB/5yEc488wzWb9+Pc9+9rP32HfzzTfz8Y9/nDvuuIMNGzbwhS98gR/96Ef88Ic/5HOf+xwbN24EYPPmzZx++un09fXx1Kc+lc985jM8/PDDnHbaaXzlK19h06ZN7Nq1i89+9rNj3sfs2bM544wz+OhHP8qtt97Ka17zmsl+a/Ze3yq4bins2Apk6fG6pZNK/vb2+1/p8ssv581vfvNj2xHBm970Jv74j/+YSy+9dNTnNOvPz8SvRlau3czQzj27gId27mbl2s0FRSRJktQ4ivwsNdoqYhExZvtIN910EyeffDIA733ve/fYd/TRRz9Wfv/GG2/khBNO4MADD+Sggw7ixBNPfKxXZ9asWSxcuBCA97znPdx4441s3ryZOXPm8MIXvhCA973vfXzve9+bxJ3W0LrlsHNoz7adQ6X2fbS33/9h3/72t7n88sv5xCceXyzgpptu4pZbbuEb3/gGF1988ajfv2b9+Zn41ci2wSHevt+N3DhtKXc/5RRunLaUt+93I9sGh578yZIkSS1urM9M9fgs1d3dzdatWx/b7u/vZ+bMmWO2j2ashOTAAw987Ovxlqke+fyxEs9h+++/P48++ihQWjajcDvGGF45VvtemMj3v6+vjw9+8IN87Wtf4xnPeMZj7cPHP/OZz+SEE07g5ptvHvX5zfjzM/GrkfcddDMr2i+je7/72S+ge7/7WdF+Ge87aPQXlyRJkh43s7NjQu3V9Pa3v50rrriCzOSHP/wh06dP5znPeQ5HHXUUd911Fz/72c945JFHuPrqq3n729/+hOcvXLiQq6++GihVmBzLa1/7WtasWcPvf/97HnroIa655prHhvPde++9/OAHPwDgy1/+Mq9+9as5/PDDueeee9iyZQsAV155Ja973euA0rDADRs2APDVr371sWscfPDBPPjgg1X4rkzQ9O6Jte+Fvf3+33vvvZx44olceeWVj/WuATz00EOPfS8eeughvvWtb41a3bVZf34mfjXyl+1f4Y/ikT3a/ige4S/bv1JQRJIkSY1j2aK5dLS37dHW0d7GskVzJ3Xek08+mVe+8pVs3ryZ7u5uLr/8cgAuueQSLrnkEgDe8pa38PznP59DDz2UP/uzP+Mzn/kMUOqVueiii1i0aBEvetGLWLJkCS95yUuecI1Pf/rTXHzxxRx11FHs2LFjzFhe9rKXcdppp3H00Ufz8pe/nA9+8IMsWLAAgBe96EV86UtfYv78+fz617/mzDPP5IADDuALX/gC73rXu5g3bx777bcfZ5xxBgB/8zd/w0c+8hFe85rX0Nb2+PftuOOO45prrql/cZdjzoX2EUl6e0epfR+N9/2v/PktX76cBx54gA996EN7LNvwy1/+kle/+tW89KUv5eijj+atb30rxx577BOu06w/vxivy7HR9PT05GTKslbVeZ3AaN/bgPMG6x2NJElS4e68805e9KIX7fXxazYOsHLtZrYNDjGzs4Nli+ayeEFXDSOcGu655x7e9ra3cdtttxUdyuT0rSrN6dvRX+rpO+ZcmL+k6Khqrp4/v9F+pyJiQ2Y+YZHC/WseTaua3l2uYjRKuyRJkp7U4gVdLZHoNa35S1oi0WsUDvWslRp0b0uSJKn5zZ49u/F7+1rYVP35mfjVyvwlcNyFMH0WEKXH4y70rx6SJEmS6s6hnrVk97YkSdIeMnPctdck7Z2J1mqxx0+SJEl1ccABB/DAAw9M+AOrpD1lJg888AAHHHDAXj/HHj9JkiTVRXd3N/39/Wzfvr3oUKSGd8ABB9DdvfeFI038JEmSVBft7e3MmTOn6DCkluRQT0mSJElqciZ+kiRJktTkTPwkSZIkqclFM1VViojtwM+LjmMUhwD3Fx2EmpavL9WSry/Vkq8v1ZKvL9XaVH2NPS8zZ4xsbKrEb6qKiN7M7Ck6DjUnX1+qJV9fqiVfX6olX1+qtUZ7jTnUU5IkSZKanImfJEmSJDU5E7/6uLToANTUfH2plnx9qZZ8famWfH2p1hrqNeYcP0mSJElqcvb4SZIkSVKTM/GroYg4NiI2R8SWiDi76HjUXCLinojYFBG3RkRv0fGo8UXE5yPiVxFxW0Xb0yPiPyLirvLj04qMUY1rjNfXeRExUH4fuzUi3lJkjGpcETErIr4dEXdGxO0R8ZFyu+9hmrRxXl8N9R7mUM8aiYg24P8CbwT6gfXAyZl5R6GBqWlExD1AT2ZOxfVj1IAi4rXA74ArMvOIcts/AL/OzBXlP2A9LTP/qsg41ZjGeH2dB/wuM/+xyNjU+CLiOcBzMvOWiDgY2AAsBk7D9zBN0jivryU00HuYPX61czSwJTPvzsxHgKuB4wuOSZLGlJnfA349ovl44Evlr79E6T86acLGeH1JVZGZ92XmLeWvHwTuBLrwPUxVMM7rq6GY+NVOF7C1YrufBnyBaEpL4FsRsSEiTi86GDWtZ2XmfVD6jw94ZsHxqPmcFRF95aGgDsPTpEXEbGAB8CN8D1OVjXh9QQO9h5n41U6M0ua4WlXTwsx8GfBm4MPlYVSS1Eg+C7wAOBK4D/g/xYajRhcRBwFfBf4iM39bdDxqLqO8vhrqPczEr3b6gVkV293AtoJiURPKzG3lx18B11AaXixV2y/LcxuG5zj8quB41EQy85eZuTszHwU+h+9jmoSIaKf0ofyqzFxdbvY9TFUx2uur0d7DTPxqZz1wWETMiYhpwEnAtQXHpCYREQeWJxcTEQcCbwJuG/9Z0j65Fnhf+ev3AV8rMBY1meEP5GUn4PuY9lFEBHA5cGdmfrJil+9hmrSxXl+N9h5mVc8aKpd0/RTQBnw+Mz9ecEhqEhHxfEq9fAD7A//i60uTFRFfBl4PHAL8EvgbYA2wCngucC/wrsy0QIcmbIzX1+spDZFK4B7gz4fnY0kTERGvBv4L2AQ8Wm7+a0rzsHwP06SM8/o6mQZ6DzPxkyRJkqQm51BPSZIkSWpyJn6SJEmS1ORM/CRJkiSpyZn4SZIkSVKTM/GTJEmSpCZn4idJ0ggRsTsibq34d3YVzz07Iqb0Wk+SpOazf9EBSJI0BQ1l5pFFByFJUrXY4ydJ0l6KiHsi4hMRcXP536Hl9udFxLqI6Cs/Prfc/qyIuCYiflz+96ryqdoi4nMRcXtEfCsiOgq7KUlSSzDxkyTpiTpGDPV8d8W+32bm0cBFwKfKbRcBV2TmfOAq4MJy+4XAdzPzpcDLgNvL7YcBF2fmS4BB4B01vh9JUouLzCw6BkmSppSI+F1mHjRK+z3AGzLz7ohoB36Rmc+IiPuB52TmznL7fZl5SERsB7oz8w8V55gN/EdmHlbe/iugPTP/rvZ3JklqVfb4SZI0MTnG12MdM5o/VHy9G+fcS5JqzMRPkqSJeXfF4w/KX38fOKn89X8Hbix/vQ44EyAi2iLiqfUKUpKkSv6FUZKkJ+qIiFsrtr+ZmcNLOjwlIn5E6Y+nJ5fblgKfj4hlwHbg/eX2jwCXRsSfUurZOxO4r+bRS5I0gnP8JEnaS+U5fj2ZeX/RsUiSNBEO9ZQkSZKkJmePnyRJkiQ1OXv8JEktJSJmR0RGhPPcJUktw8RPktRQImJtRCwfpf34iPiFCZ0kSU9k4idJajRfBN4bETGi/b3AVZm5q/4hVUeU+H+zJKnq/M9FktRo1gBPB14z3BARTwPeBlxR3n5rRGyMiN9GxNaIOG9vTx4RZ0fETyPiwYi4IyJOGLH/zyLizor9Lyu3z4qI1RGxPSIeiIiLyu3nRcQ/Vzx/j6GmEfGdiPh4RNwE/B54fkS8v+Iad0fEn4+I4fiIuLV8fz+NiGMj4l0RsWHEcf8zItbs7b1LkpqXiZ8kqaFk5hCwCji1onkJ8JPM/HF5+6Hy/k7grcCZEbF4Ly/xU0pJ5XTg/wX+OSKeAxAR7wLOK5/7qcDbgQciog34d+DnwGygC7h6Arf1XuB04ODyOX5FKZF9KqU1AS+oSDCPppTgLivf32uBe4BrgTkR8aKK874HuHICcUiSmpSJnySpEX0JeFdEdJS3Ty23AZCZ38nMTZn5aGb2AV8GXrc3J87Mf83MbeXnfgW4Czi6vPuDwD9k5vos2ZKZPy/vnwksy8yHMvPhzLxxAvfzxcy8PTN3ZebOzPx6Zv60fI3vAt/i8R7OPwU+n5n/UY5xIDN/kpl/AL5CKdkjIl5CKQn99wnEIUlqUiZ+kqSGU06qtgPHR8TzgaOAfxneHxEvj4hvl4dd7gDOAA7Zm3NHxKnlYZSDETEIHFHx3FmUegRHmgX8fBLzC7eOiOHNEfHDiPh1OYa37EUMUEp+TynPf3wvsKqcEEqSWpyJnySpUV1BqafvvcC3MvOXFfv+hdLQx1mZOR24BBhZDOYJIuJ5wOeAs4BnZGYncFvFc7cCLxjlqVuB545RUfQh4I8qtp89yjGPLaobEU8Bvgr8I/CscgzX70UMZOYPgUco9Q6egsM8JUllJn6SpEZ1BfDfgD+jYphn2cHArzPz4fKcuFP28pwHUkrCtgNExPsp9fgNuwz4XxHxx+UKnIeWk8WbgfuAFRFxYEQcEBELy8+5FXhtRDw3IqYD5zxJDNOAp5Rj2BURbwbeVLH/cuD9EXFMROwXEV0RcXjF/iuAi4BdExxuKklqYiZ+kqSGlJn3AN+nlKxdO2L3h4DlEfEgcC6lYjB7c847gP8D/AD4JTAPuKli/78CH6fUo/gg5QqjmbkbOA44FLgX6AfeXX7Of1Cae9cHbOBJ5txl5oPA0nLMv6GUtF5bsf9mygVfgB3Ad4HnVZziSkrJqr19kqTHRGY++VGSJKkhlAve/Ap4WWbeVXQ8kqSpwR4/SZKay5nAepM+SVKl0SahS5KkBhQR91AqArO3axZKklpETXv8IuLYiNgcEVsi4uxR9h8fEX3lstm9EfHqcvuschnuOyPi9oj4SC3jlCSpGWTm7Mx8XmZuLDoWSdLUUrM5fhHRBvxf4I2UJrmvB04uT5wfPuYg4KHMzIiYT2m9ocMj4jnAczLzlog4mNJk+MWVz5UkSZIk7Z1a9vgdDWzJzLsz8xHgauD4ygMy83f5eOY5XEKbzLwvM28pf/0gcCfQVcNYJUmSJKlp1XKOXxelRWaH9QMvH3lQRJwAnA88E3jrKPtnAwuAHz3ZBQ855JCcPXv2PgUrSZIkSY1uw4YN92fmjJHttUz8YpS2J4wrzcxrgGsi4rXA31JajLd0gtJQ0K8Cf5GZvx31IhGnA6cDPPe5z6W3t7cKoUuSJElS44mIn4/WXsuhnv3ArIrtbmDbWAdn5veAF0TEIQAR0U4p6bsqM1eP87xLM7MnM3tmzHhCYitJkiRJLa+Wid964LCImBMR04CTgGsrD4iIQyMiyl+/DJgGPFBuuxy4MzM/WcMYJUmSJKnp1WyoZ2buioizgLVAG/D5zLw9Is4o778EeAdwakTsBIaAd5crfL4aeC+wKSJuLZ/yrzPz+lrFK0mSJEnNqmbLORShp6cnneMnSZIkqVVFxIbM7BnZXtMF3CVJkiRJxTPxkyRJkqQmV8vlHCRJ0pNYs3GAlWs3s21wiJmdHSxbNJfFC7qKDqumWvGeVUd9q2DdctjRD9O74ZhzYf6SoqOSCmfiJ0lSQdZsHOCc1ZsY2rkbgIHBIc5ZvQmgaROhVrxn1VHfKrhuKewcKm3v2FraBpM/tTyHekqSVJCVazc/lgANG9q5m5VrNxcUUe214j2rjtYtfzzpG7ZzqNQutTgTP0mSCrJtcGhC7c2gFe9ZdbSjf2LtUgsx8ZMkqSAzOzsm1N4MWvGeVUfTuyfWLrUQEz9JkgqybNFcOtrb9mjraG9j2aK5BUVUe614z0Vas3GAhStuYM7ZX2fhihtYs3Gg6JBq65hzoX3EHxHaO0rtUouzuIskSQUZLmbSShUuW/Gei9KShXSGC7hY1VN6gsjMomOomp6enuzt7S06DEmSGoLLKjS3hStuYGCUuZNdnR3cdPYbCohIUj1ExIbM7BnZbo+fJGnqcP2tumnJ3qAWYyEdSZWc4ydJmhqG19/asRXIx9ff6ltVdGRNyWUVmp+FdCRVssdPkjQ1jLf+lr1+VWdvUPNbtmjuHr26UN9COg4lrp8iv9f+nBuHiZ8kaWpw/a26mtnZMer8L3uDmkeRhXQcSlw/RX6v/Tk3FhM/SdLUML27PMxzlHZVXdG9QaqPxQu6CvkAPt5QYhOC6irye13kte1pnDjn+EmSpgbX36qrxQu6OP/EeXR1dhCUKj2ef+I8PzipKhxKXD9Ffq+LuvZwT+PA4BDJ4z2NTb9O5STZ4ydJmhpcf6vuiuoNUvNzKHH9FPm9Lura9ijvGxM/SdLUMX+JiZ6aUqsNS3Mocf0U+b0u6tpF9yg36u+ziZ8kSVINtWIBjCILy7SaIr/XRV27yF7ORv59jswsOoaq6enpyd7e3qLDkCRJU1QRf6lfuOKGUT+kdnV2cNPZb6jptaVmNDL5glJPYz3mKTfC73NEbMjMnpHt9vhJ0pPpW+W8M6kJFPWX+qKHpRXG907VSJG9nI38+2ziJ0nj6VsF1y19fGHxHVtL2+AHGKnBFFUQoiULnfjeqRorqjhVI/8+u5yDJI1n3fLHP7gM2zlUapfUUIr6S/2yRXPpaG/bo63pC5343qkm1ci/z/b4SdJ4dvRPrF3SlFXUX+pbstCJ751qUo38+1zTxC8ijgU+DbQBl2XmihH7jwf+FngU2AX8RWbeuDfPlaS6mN5dGqI0WrukhlJk2fuWWzPR9041sUb9fa7ZUM+IaAMuBt4MvBg4OSJePOKwdcBLM/NI4APAZRN4riTV3jHnQvuI3oD2jlK7msqajQMsXHEDc87+OgtX3MCajQNFh6QqW7ygi/NPnEdXZwdBqQpfPaoAtiTfO6Upp5Y9fkcDWzLzboCIuBo4Hrhj+IDM/F3F8QcCubfPlaS6GC5CYGW6ptbI6zJpYhr1L/UNx/dOacqpZeLXBVT28fcDLx95UEScAJwPPBN460SeK7WyItaialnzl/hhpckVVe1Ramq+d0pTSi2resYobU9YLT4zr8nMw4HFlOb77fVzASLi9IjojYje7du373OwUiMZ7p0YGBwiebx3wqFp0r5p5HWZJEnaG7VM/PqBWRXb3cC2sQ7OzO8BL4iIQyby3My8NDN7MrNnxowZk49aagDj9U5Imrixqjo2wrpMahB9q+CCI+C8ztJj36qiI5LUYmo51HM9cFhEzAEGgJOAUyoPiIhDgZ9mZkbEy4BpwAPA4JM9V2pl9k5I1bVs0VxuvOYz/AVXMzPuZ1sewqc4iVcv+lDRoakZuJi5pCmgZolfZu6KiLOAtZSWZPh8Zt4eEWeU918CvAM4NSJ2AkPAuzMzgVGfW6tYpUZT1FpUUrNa3HYTb2u/jP13PwxAd9zPirbL2L/tpYAfzDVJ4y1mbuInqU6ilGc1h56enuzt7S06DKnmRlYghNJaVJYll/bRBUeMsebYLPjobfWPp9n1rSqu2mMR1z6vk9FLFQScN1jba0tqORGxITN7RrbXdAF3SbUxnNxZ1VOqkh39E2vXvity2GNR13Yxc0lTQC2Lu0iqocVtN3HTU5byswP+Ozc9ZSmL224qOiSpcY31AdwP5tU33rDHZr22i5lLmgJM/KRGNPxX6x1bgXz8r9ZWiZP2jR/M66fI3tWirj1/CRx3YWnoMFF6PO5C5/dJqiuHekqNyEIBUnUN/94UNe+slRQ57LHIa7uYuaSCmfhJjcj5SFL1+cG8Po45d895dlC/3tUiry1JBXOop9SInI8kqVEVOezRIZeSWpjLOUiNaGRlOij91doPMJIkqZ6KXJ5Fo3I5B6mZOB9JkrQ3/FBeV2s2DrTWUktFLs/y/7d3/1FyVVWix7/bJJgWMEEGRkknkzAgP0wa4jQRCYrIKKDyQ0YzoCKMMAyMvCDrDWOcNaCLGcc4+ERZglkR8CnywIxCgOfwwxXwB0QgHYLh10TyIEM6oIZIImKAJOz3R1WHStOdVCd9u7pufT9r9aq6p+6t2lV1c9O7zzn7aMDs8ZMkSSojR4cMqflLVvG5Gx5i/YZNm9vaRo3gSydNKW/yd+nkfgomjYfzHx76eAT03+PnHD9JkqQyauSaiS3oktuXbZH0AazfsIlLbl/WoIiGgMXmmoqJnyRJUhn5S/mQenrt+gG1l4LF5pqKiZ8kSVIZ+Uv5kNprbNuA2kvhqIsqw4druUTKsGXiJ0mSVEb+Uj6kLjh6Pz6y00Lu3mkmT7z+Y9y900w+stNCLjh6v0aHVhyXSGkqVvWUJEkqIytAD6kTR9zDh0ZdychNLwLQHs8ye8SVjBxxEFDiz7xjhudUkzDxkyRJKqsW/KW8YUsqLLh4c9LXY+SmFyuJd4t9B6XXpMukmPhJkiSpFHovqbBq7Xo+d8NDAMUnfxbTaQ1NvHahc/wkSZJUCg1dUsFiOq2hiZdJMfGTJElSKTR0SQWL6bSGJu7ZNfGTJElSKTR0SQUrXLaGJu7ZdY6fJEmSSuGCo/fbYo4fQNuoEUO3pEILFtNpOUddtOUcP2ianl0TP0mSJJVCTwGXhlT1VGto4mVSIjMbHcOg6ezszK6urkaHIUmSJEkNERGLM7Ozd7tz/CRJkiSp5BzqKWlAGrYwriRJkrabiZ+kujV0YdxWtXReU84jkCRJw0uhQz0j4piIWBYRyyNiVh+PfzwillZ/FkbEQTWPnR8Rj0TEwxFxXUSMLjJWSdvW0IVxW9HSeZXKYetWAlm5vWVmpV2SJGkACkv8ImIEcDlwLHAgcEpEHNhrtyeBIzKzA/gXYG712HHATKAzMycDI4CTi4pVUn0aujBuK1pw8ZbloqGyveDi4l976Ty4dDJ8YWzl1mRTkoYvr9mqQ5E9ftOA5Zn5RGa+DFwPnFC7Q2YuzMznqpv3ArUrH44E2iJiJPAG4OkCY5VUh4YujNuK1nUPrH2w2NMoSc3Da7bqVGTiNw5YWbPdXW3rzxnArQCZuQr4CvAU8AywLjPvKChOSXW64Oj9aBs1You2IV0Yt9WMaR9Y+2BpZE+jJGlgWvWabS/ngBWZ+EUfbX0uGhgRR1JJ/D5b3d6NSu/gJGAvYOeI+EQ/x54VEV0R0bV69epBCVxS306cOo4vnTSFcWPbCGDc2Da+dNIUC7sU5aiLYFSv3tRRbZX2IjWqp1GSNHCteM22l3O7FFnVsxsYX7PdTh/DNSOiA7gSODYz11Sb/xJ4MjNXV/e5ATgM+F7v4zNzLtW5gZ2dneVZjV4apk6cOs5Eb6j0VO8c6qqeY9qr/5n20S5JGl5a8Zq9tV5OK1/3q8gev0XAvhExKSJ2olKc5ebaHSJiAnADcGpm/qrmoaeAQyPiDRERwFHAYwXGqiY3f8kqps++k0mzfsT0Pt2XUwAAHuZJREFU2Xcyf8mqRockDYr5m6Yz/aXLmPTitUx/6TLmb5pe/Is2qqdRQ8+hUlLza8Vrdiv2cg6Cwnr8MnNjRJwL3E6lKufVmflIRJxdfXwOcBGwO3BFJb9jY2Z2ZuZ9EfED4AFgI7CEaq+e1Jtry6msGnZuN6qnUUOrZ6hUz1/Ne4ZKgd+11Exa8Zrdir2cgyAyyzM6srOzM7u6uhodRsuav2QVl9y+jKfXrmevsW1ccPR+Q5J4TZ99J6v6WE5g3Ng27pn13sJfX0OoxRYz99xWoS6d3M8vTuPh/IeHPh5JqlfvP1xBpZfzuMtK/XtBvSJicWZ29m4vco6fWkgje91cW65FtGDvhOd2i2jUHzQcKiWpWbViL+cgMPHToLjk9mWbk74e6zds4pLblxWe+O01tq3PXhHXliuZFpzI7bndAhr5Bw2HSklqZh0zSvv/f1GKLO6iFtLIngnXlmsRLdg74bndAhq5/lYrFoSQpBZm4qdB0V8PxFD0TLi2XIto1GLmDeS53QIa+QeNjhmV+TBjxgNRuXV+jCSVlsVdNCh6z/GDSs+Ev6Rq0DiRW2VkgRVJ0iDrr7iLPX4aFPZMqHD2TqiMHG4pSRoi9vhJktRILbZMiSSpWC7nIBWgUWsXSioRK9NJkoaAiZ+0nRq5dqEkSZI0EM7xk7bT1tYulCRJkoYTEz9pOzVy7UJJkiRpIEz8pO3UyLULJUmSpIEw8ZO20wVH70fbqBFbtLWNGsEFR+/XoIhUlPlLVjF99p1MmvUjps++k/lLVjU6JEmSpAGxuIu0nXoKuFjVs9ws4iNJksrAxE/aASdOHecv/yW3tSI+fveSJKlZONRTkrbCIj6SJKkMTPwkaSss4iNJksrAxE+StsIiPpIkqQyc4ydJW2ERH0mSVAYmfpK0DRbxkSRJzc7ET5I0bMxfssreVUmSCmDiJ0kaFlwzUZKk4ljcRZI0LGxtzURJkrRjTPwkScOCayZKklScQhO/iDgmIpZFxPKImNXH4x+PiKXVn4URcVDNY2Mj4gcR8V8R8VhEvLPIWCVJr5q/ZBXTZ9/JpFk/YvrsO5m/ZFXhr+maiZIkFaewxC8iRgCXA8cCBwKnRMSBvXZ7EjgiMzuAfwHm1jz2deC2zNwfOAh4rKhYJUmv6plrt2rtepJX59oVnfy5ZqIkScXZZuIXEedGxG7b8dzTgOWZ+URmvgxcD5xQu0NmLszM56qb9wLt1dd8I/Bu4Krqfi9n5trtiEGSNECNmmt34tRxfOmkKYwb20YA48a28aWTpljYRZKkQVBPVc83A4si4gHgauD2zMw6jhsHrKzZ7gbesZX9zwBurd7fG1gNfLs6/HMxcF5mvlDH60qSdkAj59q5ZqIkScXYZo9fZv4zsC+V3rfTgccj4t8i4s+3cWj09XR97hhxJJXE77PVppHA24FvZuZU4AXgNXMEq8eeFRFdEdG1evXqbb0dSdI2ONdOkqTyqWuOX7WH79fVn43AbsAPIuLft3JYNzC+ZrsdeLr3ThHRAVwJnJCZa2qO7c7M+6rbP6CSCPYV29zM7MzMzj322KOet1N6jSjK0HBL58Glk+ELYyu3S+c1OqLy8rMuPefaSZJUPtsc6hkRM4HTgGepJGgXZOaGiHgd8Djwj/0cugjYNyImAauAk4GP9XruCcANwKmZ+aue9sz8dUSsjIj9MnMZcBTw6IDfXQtqyQWQl86DW2bChuowtHUrK9sAHTMaF1cZ+Vm3hJ5rxSW3L+PptevZa2wbFxy9X3mvIZIktYDY1nS9iLgYuCoz/7uPxw7IzH6rbUbEB4CvASOAqzPzixFxNkBmzomIK4G/Anqee2NmdlaPPZhKorkT8ATwNzWFYPrU2dmZXV1dW30/ZTd99p2s6mMezrixbdwz670NiGgIXDq5koD0NmY8nP/w0MdTZn7WkiRJw1pELO7JqWrVU9zlP4Hf1TzRrsCBmXnf1pI+gMz8z+rxtW1zau6fCZzZz7EPAq8JWFvXkgsgr+seWLu2n5+1JElSU6pnjt83gT/UbL9QbdMw1JJFGca0D6xd28/PWpIkqSnVk/hF7fINmfkK9fUUqgFasijDURfBqF6J7ai2SrsGV4M/65YsXCRJkjQI6kn8noiImRExqvpzHpU5dxqGWnIB5I4ZcNxllXlmROX2uMssNlKEBn7WPYWLVq1dT/Jq4SKTP0mSpG2rp7jLnsBlwHuprMO3APhMZv62+PAGxuIuUnm1ZOEiSZKkAdru4i7VBO/kQqKSpDq1ZOEiSZKkQVLPOn6jgTOAtwGje9oz81MFxiVJW9hrbFufPX6lLlwkSZI0SOqZ43cN8GbgaOCnQDvwfJFBSVJvLVm4SJIkaZDUk/jtk5kXAi9k5neADwJTig1LkrbUkoWLJEmSBkk9yzJsqN6ujYjJwK+BiYVFJEn9OHHqOBM9SZKk7VBP4jc3InYD/hm4GdgFuLDQqCRJkiRJg2ariV9EvA74fWY+B/wM2HtIopIkSZIkDZqtzvHLzFeAc4coFkmSJElSAeop7vLjiPiHiBgfEW/q+Sk8MkmSJEnSoKhnjl/Pen2frmlLHPYpSZIkSU1hm4lfZk4aikCkprR0Hiy4GNZ1w5h2OOoi6JjR6KgkSZKkLWwz8YuIT/bVnpnfHfxwpCaydB7cMhM2rK9sr1tZ2QaTP0mSJA0r9Qz1PKTm/mjgKOABwMRPrW3Bxa8mfT02rK+0m/hJkiRpGKlnqOf/qN2OiDHANYVFJDWLdd0Da5ckSZIapJ6qnr39Edh3sAORms6Y9oG1S5IkSQ1Szxy/W6hU8YRKonggMK/IoKSmcNRFW87xAxjVVmmXJEmShpF65vh9peb+RuC/M9OxbFLPPD6rekqSJGmYqyfxewp4JjNfBIiItoiYmJkrCo1MagYdM0z0JEmSNOzVM8fvP4BXarY3VdskSZIkSU2gnsRvZGa+3LNRvb9TcSFJkiRJkgZTPYnf6og4vmcjIk4Ani0uJEmSJEnSYKon8Tsb+KeIeCoingI+C/xdPU8eEcdExLKIWB4Rs/p4/OMRsbT6szAiDur1+IiIWBIR/7ee15MkSZIkvVY9C7j/P+DQiNgFiMx8vp4njogRwOXA+4BuYFFE3JyZj9bs9iRwRGY+FxHHAnOBd9Q8fh7wGPDGut6NKpbOs9KkJEmSpM222eMXEf8WEWMz8w+Z+XxE7BYR/1rHc08DlmfmE9V5gdcDJ9TukJkLM/O56ua9wOaVryOiHfggcGW9b0ZUkr5bZsK6lUBWbm+ZWWmXJEmS1JLqGep5bGau7dmoJmofqOO4ccDKmu3ualt/zgBurdn+GvCPbFlRVNuy4OItFxSHyvaCixsTjyRJkqSGqyfxGxERr+/ZiIg24PVb2X/zrn20ZZ87RhxJJfH7bHX7Q8BvM3PxNl8k4qyI6IqIrtWrV9cRVsmt6x5YuyRJkqTSqyfx+x6wICLOiIgzgB8D36njuG5gfM12O/B0750iooPKcM4TMnNNtXk6cHxErKAyRPS9EfG9vl4kM+dmZmdmdu6xxx51hFVyY9oH1i5JkiSp9LaZ+GXmvwP/ChwAHAjcBvxZHc+9CNg3IiZFxE7AycDNtTtExATgBuDUzPxVzWt+LjPbM3Ni9bg7M/MT9b2lFnfURTCqbcu2UW2V9qItnQeXToYvjK3cOq9QkiRJGha2WdWz6tdU5trNoFKJ84fbOiAzN0bEucDtwAjg6sx8JCLOrj4+B7gI2B24IiIANmZm54DfhV7VU71zqKt69hSV6Zlf2FNUpjYmSZIkSQ0RmX1OuyMi3kqlt+0UYA3wfeAfMrOe3r6G6OzszK6urkaH0ZounVytJNrLmPFw/sNDH48kSZLUgiJicV+daVvr8fsv4OfAcZm5vPok5xcUn5qdRWUkSZKkYWtrc/z+isoQz7si4lsRcRR9V+qULCojSZIkDWP9Jn6ZeWNm/jWwP/AT4HzgTyPimxHx/iGKT82ikUVlJEmSJG1VPVU9X8jMazPzQ1SWZHgQmFV4ZGouHTPguMsqc/qIyu1xl1nYRZIkSRoG+i3u0ows7iJJkiSplfVX3KWeBdwlSZIkSU3MxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSs7ET5IkSZJKzsRPkiRJkkrOxE+SJEmSSq7QxC8ijomIZRGxPCJm9fH4xyNiafVnYUQcVG0fHxF3RcRjEfFIRJxXZJySJEmSVGYji3riiBgBXA68D+gGFkXEzZn5aM1uTwJHZOZzEXEsMBd4B7AR+J+Z+UBE7Aosjogf9zpWkiRJklSHInv8pgHLM/OJzHwZuB44oXaHzFyYmc9VN+8F2qvtz2TmA9X7zwOPAeMKjFWSJEmSSqvIxG8csLJmu5utJ29nALf2boyIicBU4L5BjE2SJEmSWkZhQz2B6KMt+9wx4kgqid/hvdp3AX4IfCYzf9/PsWcBZwFMmDBhR+KVJEmSpFIqssevGxhfs90OPN17p4joAK4ETsjMNTXto6gkfddm5g39vUhmzs3Mzszs3GOPPQYteEmSJEkqiyITv0XAvhExKSJ2Ak4Gbq7dISImADcAp2bmr2raA7gKeCwzv1pgjJIkSZJUeoUN9czMjRFxLnA7MAK4OjMfiYizq4/PAS4CdgeuqOR6bMzMTmA6cCrwUEQ8WH3Kf8rM/ywqXkmSJEkqq8jsc9pdU+rs7Myurq5GhyFJkiRJDRERi6udaVsodAF3SZIkSVLjmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyZn4SZIkSVLJmfhJkiRJUsmZ+EmSJElSyRWa+EXEMRGxLCKWR8SsPh7/eEQsrf4sjIiD6j1WkiRJklSfwhK/iBgBXA4cCxwInBIRB/ba7UngiMzsAP4FmDuAYyVJkiRJdSiyx28asDwzn8jMl4HrgRNqd8jMhZn5XHXzXqC93mMlSZIkSfUpMvEbB6ys2e6utvXnDODW7TxWkiRJktSPkQU+d/TRln3uGHEklcTv8O049izgLIAJEyYMPEpJkiRJKrkie/y6gfE12+3A0713iogO4ErghMxcM5BjATJzbmZ2ZmbnHnvsMSiBS5IkSVKZFJn4LQL2jYhJEbETcDJwc+0OETEBuAE4NTN/NZBjJUmSJEn1KWyoZ2ZujIhzgduBEcDVmflIRJxdfXwOcBGwO3BFRABsrPbe9XlsUbFKkiRJUplFZp9T55pSZ2dndnV1NTqMVy2dBwsuhnXdMKYdjroIOmY0OipJkiRJJRURizOzs3d7kcVdWtvSeXDLTNiwvrK9bmVlG0z+JEmSJA2pIuf4tbYFF7+a9PXYsL7SLkmSJElDyMSvKOu6B9YuSZIkSQUx8SvKmPaBtUuSJElSQUz8inLURTCqbcu2UW2VdkmSJEkaQiZ+RemYAcddBmPGA1G5Pe4yC7tIkiRJGnJW9SxSxwwTPUmSJEkNZ4+fJEmSJJWciZ8kSZIklZyJnyRJkiSVnHP8JEmSNCQ2bNhAd3c3L774YqNDkZre6NGjaW9vZ9SoUXXtb+InSZKkIdHd3c2uu+7KxIkTiYhGhyM1rcxkzZo1dHd3M2nSpLqOcainJEmShsSLL77I7rvvbtIn7aCIYPfddx9Q77mJnyRJkoaMSZ80OAb6b8nET5IkSS3jU5/6FHvuuSeTJ0/ud5/MZObMmeyzzz50dHTwwAMPbH7stttuY7/99mOfffZh9uzZdb3mLrvsssNxb6+f/OQnLFy4sGGvP9jq+fyvvfZaOjo66Ojo4LDDDuOXv/zl5scmTpzIlClTOPjgg+ns7KzrNcvy/Zn4SZIkaViav2QV02ffyaRZP2L67DuZv2TVDj/n6aefzm233bbVfW699VYef/xxHn/8cebOncs555wDwKZNm/j0pz/NrbfeyqOPPsp1113Ho48+ul1xbNq0abuOG6iGJn5L58Glk+ELYyu3S+ft0NPV+/lPmjSJn/70pyxdupQLL7yQs846a4vH77rrLh588EG6urp2KJahYOInSZKkUpu/ZBWfu+EhVq1dTwKr1q7nczc8tMPJ37vf/W7e9KY3bXWfm266iU9+8pNEBIceeihr167lmWee4f7772efffZh7733ZqedduLkk0/mpptues3xTz75JO985zs55JBDuPDCCze3/+QnP+HII4/kYx/7GFOmTAHgq1/9KpMnT2by5Ml87WtfA2DFihXsv//+nHbaaXR0dPCRj3yEP/7xjwAsWLCAqVOnMmXKFD71qU/x0ksvAZWerGeffRaArq4u3vOe97BixQrmzJnDpZdeysEHH8zPf/7zHfrsBmTpPLhlJqxbCWTl9paZO5T81fv5H3bYYey2224AHHrooXR3dw/odcr6/Zn4SZIkadi55PZlrN+wZa/K+g2buOT2ZYW/9qpVqxg/fvzm7fb2dlatWtVve2/nnXce55xzDosWLeLNb37zFo/df//9fPGLX+TRRx9l8eLFfPvb3+a+++7j3nvv5Vvf+hZLliwBYNmyZZx11lksXbqUN77xjVxxxRW8+OKLnH766Xz/+9/noYceYuPGjXzzm9/s931MnDiRs88+m/PPP58HH3yQd73rXTv60dRvwcWwYf2WbRvWV9q3U72ff62rrrqKY489dvN2RPD+97+fv/iLv2Du3Ll9HlPW78/ET5IkScPO02vXD6h9MGXma9oiot/23u655x5OOeUUAE499dQtHps2bdrm8vt33303H/7wh9l5553ZZZddOOmkkzb36owfP57p06cD8IlPfIK7776bZcuWMWnSJN761rcCcNppp/Gzn/1sB95pgdb108vWX3sd6v38e9x1111cddVVfPnLX97cds899/DAAw9w6623cvnll/f5+ZX1+zPxK1AR49IlSZJawV5j2wbUPpja29tZuXLl5u3u7m722muvftv70l9CsvPOO2++31ci09/x/SWePUaOHMkrr7wCMKAS/4UZ0z6w9joM5PNfunQpZ555JjfddBO777775vae/ffcc08+/OEPc//99/d5fBm/PxO/ghQ1Ll2SJKkVXHD0frSNGrFFW9uoEVxw9H6Fv/bxxx/Pd7/7XTKTe++9lzFjxvCWt7yFQw45hMcff5wnn3ySl19+meuvv57jjz/+NcdPnz6d66+/HqhUmOzPu9/9bubPn88f//hHXnjhBW688cbNw/meeuopfvGLXwBw3XXXcfjhh7P//vuzYsUKli9fDsA111zDEUccAVSGBS5evBiAH/7wh5tfY9ddd+X5558fhE9lgI66CEb1StJHtVXat1O9n/9TTz3FSSedxDXXXLO5dw3ghRde2PxZvPDCC9xxxx19Vnct6/dn4leQRo5LlyRJanYnTh3Hl06awrixbQQwbmwbXzppCidOHbdDz3vKKafwzne+k2XLltHe3s5VV10FwJw5c5gzZw4AH/jAB9h7773ZZ599+Nu//VuuuOIKoNIr841vfIOjjz6aAw44gBkzZvC2t73tNa/x9a9/ncsvv5xDDjmEdevW9RvL29/+dk4//XSmTZvGO97xDs4880ymTp0KwAEHHMB3vvMdOjo6+N3vfsc555zD6NGj+fa3v81HP/pRpkyZwute9zrOPvtsAD7/+c9z3nnn8a53vYsRI15NmI877jhuvPHGoS/u0jEDjrsMxowHonJ73GWV9u20tc+/9vu7+OKLWbNmDX//93+/xbINv/nNbzj88MM56KCDmDZtGh/84Ac55phjXvM6Zf3+Ymtdjs2ms7Mzd6Qs62CaNOtH9PXJBvDk7A8OdTiSJEkN99hjj3HAAQc0Ooxhb8WKFXzoQx/i4YcfbnQo2g5D+f319W8qIhZn5msWKbTHryCNHJcuSZIkSbVM/ArSyHHpkiRJal4TJ060t6+JDdfvr9DELyKOiYhlEbE8Imb18fj+EfGLiHgpIv6h12PnR8QjEfFwRFwXEaOLjHWwFTUuXZIkSZIGamRRTxwRI4DLgfcB3cCiiLg5Mx+t2e13wEzgxF7Hjqu2H5iZ6yNiHnAy8L+LircIJ04dZ6InSZJUIzO3uvaapPoMtFZLkT1+04DlmflEZr4MXA+cULtDZv42MxcBG/o4fiTQFhEjgTcATxcYqyRJkgo2evRo1qxZM+BfWCVtKTNZs2YNo0fXPyiysB4/YBywsma7G3hHPQdm5qqI+ArwFLAeuCMz7xj8ECVJkjRU2tvb6e7uZvXq1Y0ORWp6o0ePpr29ve79i0z8+urDr+vPOxGxG5XewUnAWuA/IuITmfm9PvY9CzgLYMKECdsfrSRJkgo1atQoJk2a1OgwpJZU5FDPbmB8zXY79Q/X/EvgycxcnZkbgBuAw/raMTPnZmZnZnbuscceOxSwJEmSJJVRkYnfImDfiJgUETtRKc5yc53HPgUcGhFviMrs36OAxwqKU5IkSZJKrbChnpm5MSLOBW4HRgBXZ+YjEXF29fE5EfFmoAt4I/BKRHyGSiXP+yLiB8ADwEZgCTC3qFglSZIkqcyiTFWVImI18N+NjqMPfwI82+ggVFqeXyqS55eK5PmlInl+qWjD9Rz7s8x8zRy4UiV+w1VEdGVmZ6PjUDl5fqlInl8qkueXiuT5paI12zlW5Bw/SZIkSdIwYOInSZIkSSVn4jc0LEyjInl+qUieXyqS55eK5PmlojXVOeYcP0mSJEkqOXv8JEmSJKnkTPwKFBHHRMSyiFgeEbMaHY/KJSJWRMRDEfFgRHQ1Oh41v4i4OiJ+GxEP17S9KSJ+HBGPV293a2SMal79nF9fiIhV1evYgxHxgUbGqOYVEeMj4q6IeCwiHomI86rtXsO0w7ZyfjXVNcyhngWJiBHAr4D3Ad3AIuCUzHy0oYGpNCJiBdCZmcNx/Rg1oYh4N/AH4LuZObna9u/A7zJzdvUPWLtl5mcbGaeaUz/n1xeAP2TmVxoZm5pfRLwFeEtmPhARuwKLgROB0/Eaph20lfNrBk10DbPHrzjTgOWZ+URmvgxcD5zQ4JgkqV+Z+TPgd72aTwC+U73/HSr/0UkD1s/5JQ2KzHwmMx+o3n8eeAwYh9cwDYKtnF9NxcSvOOOAlTXb3TThCaJhLYE7ImJxRJzV6GBUWn+amc9A5T8+YM8Gx6PyOTcillaHgjoMTzssIiYCU4H78BqmQdbr/IImuoaZ+BUn+mhzXK0G0/TMfDtwLPDp6jAqSWom3wT+HDgYeAb4X40NR80uInYBfgh8JjN/3+h4VC59nF9NdQ0z8StONzC+ZrsdeLpBsaiEMvPp6u1vgRupDC+WBttvqnMbeuY4/LbB8ahEMvM3mbkpM18BvoXXMe2AiBhF5ZfyazPzhmqz1zANir7Or2a7hpn4FWcRsG9ETIqInYCTgZsbHJNKIiJ2rk4uJiJ2Bt4PPLz1o6TtcjNwWvX+acBNDYxFJdPzC3nVh/E6pu0UEQFcBTyWmV+techrmHZYf+dXs13DrOpZoGpJ168BI4CrM/OLDQ5JJRERe1Pp5QMYCfwfzy/tqIi4DngP8CfAb4DPA/OBecAE4Cngo5lpgQ4NWD/n13uoDJFKYAXwdz3zsaSBiIjDgZ8DDwGvVJv/ico8LK9h2iFbOb9OoYmuYSZ+kiRJklRyDvWUJEmSpJIz8ZMkSZKkkjPxkyRJkqSSM/GTJEmSpJIz8ZMkSZKkkjPxkySpl4jYFBEP1vzMGsTnnhgRw3qtJ0lS+YxsdACSJA1D6zPz4EYHIUnSYLHHT5KkOkXEioj4ckTcX/3Zp9r+ZxGxICKWVm8nVNv/NCJujIhfVn8Oqz7ViIj4VkQ8EhF3RERbw96UJKklmPhJkvRabb2Gev51zWO/z8xpwDeAr1XbvgF8NzM7gGuBy6rtlwE/zcyDgLcDj1Tb9wUuz8y3AWuBvyr4/UiSWlxkZqNjkCRpWImIP2TmLn20rwDem5lPRMQo4NeZuXtEPAu8JTM3VNufycw/iYjVQHtmvlTzHBOBH2fmvtXtzwKjMvNfi39nkqRWZY+fJEkDk/3c72+fvrxUc38TzrmXJBXMxE+SpIH565rbX1TvLwROrt7/OHB39f4C4ByAiBgREW8cqiAlSarlXxglSXqttoh4sGb7tszsWdLh9RFxH5U/np5SbZsJXB0RFwCrgb+ptp8HzI2IM6j07J0DPFN49JIk9eIcP0mS6lSd49eZmc82OhZJkgbCoZ6SJEmSVHL2+EmSJElSydnjJ0mSJEklZ+InSZIkSSVn4idJkiRJJWfiJ0mSJEklZ+InSZIkSSVn4idJkiRJJff/AXSqXswNxk2KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
